Apparatuses, systems, and methods for providing a visual program for machine vision systems

ABSTRACT

Techniques include systems, computerized methods, and computer readable media for creating a graphical program in a graphical program development environment. A spreadsheet node having an input terminal in the graphical program is instantiated. The spreadsheet node is associated with a spreadsheet that specifies a list of functions to be executed in a computing device, and the input terminal is connected to the first terminal of the first node, indicating a data connection between the first terminal of the first node and the input terminal of the spreadsheet node. The input terminal of the spreadsheet node is associated with a first cell in the spreadsheet, indicating that the first cell in the spreadsheet be populated with any data received by the input terminal. A human readable file is generated specifying the graphical program, including the spreadsheet node.

TECHNICAL FIELD

Disclosed herein are apparatus, systems, and methods that relate toprogramming a machine vision system using a graphical program.

BACKGROUND

A machine vision system is becoming a popular tool to replace humanvision in a wide range of applications, such as manufacturingoperations. A machine vision system typically provides automated,computer-based image acquisition and analysis capabilities that can beemployed for tasks such as measurement and inspection of parts ormaterials. For such tasks, a machine vision system is typicallyconfigured with a camera for acquiring an image of an object ofinterest, e.g., a part being produced, and further is configured withprocessing functionality to process the acquired image and produceinformation about the object. The camera and/or the processingfunctionality can be programmed to adapt the operation of the machinevision system, including the camera and/or the processing functionality,to the applications of interest.

As machine vision systems become popular in various applications, it isoften desirable for machine vision systems to be used and programmed byusers with little (or no) programming expertise. Therefore, it isdesirable to provide a program development environment for machinevisions systems that is suited to a large spectrum of users.

SUMMARY

In accordance with the disclosed subject matter, apparatus, systems, andmethods are provided for programming a machine vision system using agraphical program.

Some embodiments include a computer-implemented method for creating agraphical program in a graphical program development environment. Themethod includes providing the graphical program in the graphical programdevelopment environment, wherein the graphical program comprises a firstnode having a first terminal. The method includes instantiating aspreadsheet node having an input terminal in the graphical program,wherein the spreadsheet node is associated with a spreadsheet thatspecifies a list of functions to be executed in a computing device, andthe input terminal is connected to the first terminal of the first node,indicating a data connection between the first terminal of the firstnode and the input terminal of the spreadsheet node. The method includesassociating the input terminal of the spreadsheet node with a first cellin the spreadsheet, indicating that the first cell in the spreadsheet bepopulated with any data received by the input terminal. The methodincludes generating a human readable file specifying the graphicalprogram including the spreadsheet node.

Some embodiments include a system for creating a graphical program in agraphical program development environment. The system includes aprocessor in communication with a memory, wherein the processor isconfigured to run a computer program stored in the memory that isconfigured to provide the graphical program in the graphical programdevelopment environment, wherein the graphical program comprises a firstnode having a first terminal. The program is configured to instantiate aspreadsheet node having an input terminal in the graphical program,wherein the spreadsheet node is associated with a spreadsheet thatspecifies a list of functions to be executed in a computing device, andthe input terminal is connected to the first terminal of the first node,indicating a data connection between the first terminal of the firstnode and the input terminal of the spreadsheet node. The program isconfigured to associate the input terminal of the spreadsheet node witha first cell in the spreadsheet, indicating that the first cell in thespreadsheet be populated with any data received by the input terminal.The program is configured to generate a human readable file specifyingthe graphical program including the spreadsheet node.

Some embodiments include a non-transitory computer readable mediumhaving executable instructions associated with a system for creating agraphical program in a graphical program development environment. Theinstructions are operable to cause the system to provide the graphicalprogram in the graphical program development environment, wherein thegraphical program comprises a first node having a first terminal. Theinstructions are operable to cause the system to instantiate aspreadsheet node having an input terminal in the graphical program,wherein the spreadsheet node is associated with a spreadsheet thatspecifies a list of functions to be executed in a computing device, andthe input terminal is connected to the first terminal of the first node,indicating a data connection between the first terminal of the firstnode and the input terminal of the spreadsheet node. The instructionsare operable to cause the system to associate the input terminal of thespreadsheet node with a first cell in the spreadsheet, indicating thatthe first cell in the spreadsheet be populated with any data received bythe input terminal. The instructions are operable to cause the system togenerate a human readable file specifying the graphical programincluding the spreadsheet node.

In some embodiments, the human readable file is transferred to acontroller that is configured to execute the graphical program specifiedin the human readable file. An execution property of the spreadsheetnode can be received indicating the computing device designated toexecute the computer-readable instructions of the spreadsheet node. Anexecution property of the first node can be received indicating that thecomputing device is designated to execute computer-readable instructionsof the first node. It can be determined, at the controller, based on theexecution property of the spreadsheet node, that the computing device isdesignated to execute the computer-readable instructions of thespreadsheet node, and the computer-readable instructions of thespreadsheet node can be transmitted to the computing device forexecution of the computer-readable instructions of the spreadsheet node.

In some embodiments, upon receiving a request to view the spreadsheetassociated with the spreadsheet node, the spreadsheet in the graphicalprogram development environment is displayed, first data to beassociated with a second cell in the spreadsheet is received, and thesecond cell in the spreadsheet is populated using the first data. Thecomputing device can be a camera. The graphical program can beconfigured to perform a machine vision task. The computer-readableinstructions of the spreadsheet node can be designed to perform machinevision analysis of an image. The input terminal of the spreadsheet nodecan be configured to receive the image from the first terminal of thefirst node.

In some embodiments, a second spreadsheet node is provided in thegraphical program, wherein the second spreadsheet node is associatedwith a second spreadsheet designed to be executed at another computingdevice. The spreadsheet node can include an output terminal, wherein theoutput terminal is associated with a second cell in the spreadsheetindicative of an output of the computer-readable instructions of thespreadsheet node. The output terminal of the spreadsheet node can beconnected to a terminal of a second node of the graphical program toprovide the output to the terminal of the second node of the graphicalprogram.

In some embodiments, a visualization interface is provided having agraphical element, and associating the graphical element with a secondcell in the spreadsheet to display a content of the second cell at thegraphical element. The content of the second cell can include an image,and wherein associating the graphical element with the second cell caninclude assigning, to the graphical element, a pointer that referencesthe image. A portion of the spreadsheet can be displayed in thegraphical element of the visualization interface. A request to modify acell of the spreadsheet can be received using the visualizationinterface.

In some embodiments, the computer-readable instructions of thespreadsheet node are executed at the computing device, wherein executingthe computer-readable instructions includes receiving a pointer to firstdata to be processed by the computing device, determining whether thecomputing device maintains the first data locally at the computingdevice, upon determining that the first data is maintained locally atthe computing device, processing the first data, and upon determiningthat the first data is not maintained locally at the computing device,retrieving the first data from a storage medium referenced by thepointer and processing the first data. The computing device can includea plurality of computing modules, and wherein the method furthercomprises automatically selecting one or more of the computing modulesto execute the computer-readable instructions of the spreadsheet node.Automatically selecting the one or more of the computing modules caninclude determining a computation load of the one or more of thecomputing modules.

There has thus been outlined, rather broadly, the features of thedisclosed subject matter in order that the detailed description thereofthat follows may be better understood, and in order that the presentcontribution to the art may be better appreciated. There are, of course,additional features of the disclosed subject matter that will bedescribed hereinafter and which will form the subject matter of theclaims appended hereto. It is to be understood that the phraseology andterminology employed herein are for the purpose of description andshould not be regarded as limiting.

BRIEF DESCRIPTION OF THE DRAWINGS

Various objects, features, and advantages of the disclosed subjectmatter can be more fully appreciated with reference to the followingdetailed description of the disclosed subject matter when considered inconnection with the following drawings, in which like reference numeralsidentify like elements.

FIG. 1 shows an exemplary visual program development environment for amachine vision system, in accordance with some embodiments;

FIG. 2 shows an exemplary visual program developed in a graphicalenvironment, in accordance with some embodiments;

FIG. 3 shows an exemplary graphical interface for programming a visualprogram, in accordance with some embodiments;

FIG. 4 shows a spreadsheet interface shown in the graphical interface,in accordance with some embodiments;

FIG. 5 shows a spreadsheet interface shown in the graphical interface,in accordance with some embodiments;

FIG. 6 shows an exemplary program panel of a graphical interface for avisual program with multiple cameras, in accordance with someembodiments;

FIG. 7 shows an exemplary program panel of a graphical interface for avisual program with a single task with multiple nodes (that each use aspreadsheet) for the same camera, in accordance with some embodiments;

FIG. 8 shows an exemplary program panel of a graphical interface for avisual program with multiple nodes (that each use a spreadsheet) withmultiple cameras, according to some embodiments;

FIG. 9 shows an exemplary menu for publishing an input and/or an outputof a spreadsheet, according to some embodiments;

FIGS. 10A-10C show an exemplary object for a spreadsheet node, accordingto some embodiments; and

FIG. 11 is an exemplary list of objects for cells in a spreadsheet,according to some embodiments.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forthregarding the systems and methods of the disclosed subject matter andthe environment in which such systems and methods may operate, etc., inorder to provide a thorough understanding of the disclosed subjectmatter. It will be apparent to one skilled in the art, however, that thedisclosed subject matter may be practiced without such specific details,and that certain features, which are well known in the art, are notdescribed in detail in order to avoid complication of the disclosedsubject matter. In addition, it will be understood that the examplesprovided below are exemplary, and that it is contemplated that there areother systems and methods that are within the scope of the disclosedsubject matter.

As machine vision systems become popular in various applications, it isoften desirable for machine vision systems to be used and programmedeven by users without programming expertise. Therefore, it is desirableto provide a program development environment for machine vision systemsthat is suited to a large spectrum of users.

One such exemplary program development environment is a visual programdevelopment (VPD) environment. A VPD environment allows a user todevelop portions of a computer program, or even an entire computerprogram, by manipulating visual elements graphically, rather than byspecifying them textually.

A visual program, developed by a VPD environment, can graphicallyrepresent a dataflow by using various nodes and interconnections betweenthose nodes. The nodes can be represented by boxes and/or other shapesin the visual program. The interconnections can be represented by linesin the visual program. Each node can represent, for example, an elementof the machine vision system (e.g., a camera) or an operation performedby the machine visions system (e.g., an acquire action).

The visual program can configure inputs and outputs for each node. Forexample, each node can include one or more input terminals to configuredata input(s) to the node, and/or one or more output terminals toconfigure data output(s) from the node. The inputs and outputs can berepresented graphically in the visual program, such as by using visualobjects extending from a side of the node (e.g., such as a small circleconnected to the node by a small line, as shown in FIGS. 2 and 3).

The interconnections can be used to connect inputs and outputs of thevarious nodes of the graphical program to build the graphical program,indicating a data flow between nodes. For example, an output terminal ofa first node can be connected to an input terminal of a second node viaa first interconnection to configure the program so that the second nodereceives (as input) the output data from the output terminal of thefirst node. Similarly, an output terminal of the second node can beconnected to an input terminal of a third node via a secondinterconnection to configure the graphical program such that the secondnode provides data to the input terminal of the third node.

A user may assemble a visual program by placing one or more nodes intothe VPD environment and connecting the one or more nodes usinginterconnections. For example, if a user wants to perform an addition oftwo numbers, the user can select a node corresponding to an additionoperation and place the node into the VPD environment. The “addition”node can have two input terminals configured to receive the two numbersthat should be added at the “addition node” (also known as operands) andone output terminal to provide the result of the addition operation.Therefore, the user can provide, using the interconnections, the twonumbers to the input terminals of the “addition node” to perform theaddition. The output terminal will contain the result of the addition. Auser can use the output of the “addition node” to further process theresult of the addition in the graphical program.

One potential weakness of a VPD environment is that when needing todesign complex applications—such as a machine vision application—thegamut of operations that can be performed by the nodes can be limited.For example, the VPD environment may only be configured to provide alimited set of node types dedicated to a limited variety of operations,such as additions, subtractions, and multiplications. Therefore,situations can occur when a user desires to perform a particularoperation but the operation is not offered by the VPD environment.Therefore, the flexibility and customizability of the programs developedin the VPD environment can be unintentionally limited.

Some VPD environments attempt to improve the flexibility by providing anode representing a custom text program. For example, a node, sometimesreferred to as a “script” node, can be associated with a script programthat is designed to perform a specific operation. Therefore, when a userwants to deploy a node that can perform a particular operation that isnot provided by the VPD environment, the user can develop an actualsoftware (or script) program for that particular operation, andassociate the script program with a script node so that the particularcustom operation can be performed by the script node.

Unfortunately, most novice users fail to leverage the flexibilityprovided by a script node because the use of a script node requires theability to write the script program. In some sense, the use of a scriptnode defeats the purpose of the VPD environment because the purpose ofthe VPD environment was to provide a program development environment inwhich a program can be developed even without expertise in writingprograms.

The techniques described herein provide for a new programming mechanismfor improving the flexibility of a VPD environment. In some embodiments,the VPD environment can include a spreadsheet node. As described furtherherein, a spreadsheet node can be associated with a spreadsheet thatincludes one or more cells that are configured to execute variousoperations. For example, a spreadsheet node can be used to specify aparticular way in which an input image is processed; as another example,a spreadsheet node can be used to specify a particular way in which aninput matrix is manipulated to create an output matrix.

A benefit of using a spreadsheet node, as opposed to a script node, isthat a spreadsheet associated with a spreadsheet node can be programmedeasily. For example, a user can use a spreadsheet program to build aspreadsheet designed to carry out a particular function. The spreadsheetprogram for the spreadsheet node can be designed, as described herein,so that it is easy to learn and simple to manipulate so that even novicecomputer users can easily program the spreadsheet. Furthermore, aspreadsheet program can provide built-in functions that can be easilycombined or instantiated to carry out a desired operation in thespreadsheet. Thus, compared to a script node that often requiressignificant programming expertise, a spreadsheet node can strike abetter balance between the flexibility of the node and the easy usage oraccessibility of the node.

FIG. 1 illustrates an exemplary computing system that enables the use ofa VPD environment in accordance with some embodiments. The system 100can include a host device 102, a central controller 104, one or morecomputing devices 106, and one or more display devices 108.

In some embodiments, the host device 102 can include a processor 110, amemory device 112, and a visual program development (VPD) environmentmodule 114. The processor 104 can execute instructions and one or morememory devices 106 can store instructions and/or data. The memory device106 can be a non-transitory computer readable medium, such as a dynamicrandom access memory (DRAM), a static random access memory (SRAM), flashmemory, a magnetic disk drive, an optical drive, a programmableread-only memory (PROM), a read-only memory (ROM), or any other memoryor combination of memories. The memory device 106 can be used totemporarily store data. The memory device 106 can also be used forlong-term data storage. The processor 104 and the memory device 106 canbe supplemented by and/or incorporated into special purpose logiccircuitry.

The VPD environment module 114 can be configured to provide a graphicalenvironment in which a user can develop a visual program, as describedfurther herein. Therefore, a user can use the graphical environment ofthe VPD environment module 114 to develop a visual program for thecomputing system 100. Once the user defines the visual program, thevisual program can be launched in the computing system 100. For example,the VPD environment module 114 can package the visual program as aproject file, and provide the project file to the central controller 104via an interface 116 so that the central controller 104 can launch orexecute the visual program in the project file.

In some embodiments, the central controller 104 can include a processor118, a memory device 120, and a visual program execution (VPE) module122. The VPE module 122 can be configured to “un-pack” the project fileto retrieve the visual program, and prepare the visual program forexecution. To this end, the VPD module 122 can retrieve pre-existingmachine readable programs as defined in the visual program, as describedfurther below. Depending on the definition of the visual program, thecentral controller 104 can assign one or more operations to one or morecomputing devices 106. The computing devices 106 can be, for example,cameras and/or smart cameras. A smart camera can include, for example, aprocessor and a memory such that the smart camera can be configured toexecute one or more portions of the program designed using the VPDenvironment 114. Also, depending on the definition of the visualprogram, the central controller 104 can cause one or more displaydevices 108 to display data. In some embodiments, the central controller104 can be a part of the host device 102. For example, the VPE module122 can be a part of the host device 102.

In some embodiments, the interface 116 can be implemented in hardware tosend and receive signals in a variety of mediums, such as optical,copper, and/or wireless interfaces, and in a number of differentprotocols, some of which may be non-transient.

In some embodiments, the VPD environment module 114 and the VPE module112 can be implemented in software. The software can run on a processor104 or a processor 118 capable of executing computer instructions orcomputer code. The processor 104 and the processor 118 can beimplemented in hardware using an application specific integrated circuit(ASIC), programmable logic array (PLA), digital signal processor (DSP),field programmable gate array (FPGA), or any other integrated circuit.The processor 104 and the processor 118 suitable for the execution of acomputer program include, by way of example, both general and specialpurpose microprocessors, digital signal processors, and any one or moreprocessors of any kind of digital computer. Generally, the processor 104and the processor 118 receive instructions and data from a read-onlymemory or a random access memory or both.

The VPD environment module 114 and the VPE module 112 can be implementedin digital electronic circuitry, or in computer hardware, firmware,software, or in combinations of them. The implementation can be as acomputer program product, e.g., a computer program tangibly embodied ina machine-readable storage device, for execution by, or to control theoperation of, a data processing apparatus, e.g., a programmableprocessor, a computer, and/or multiple computers. A computer program canbe written in any form of computer or programming language, includingsource code, compiled code, interpreted code and/or machine code, andthe computer program can be deployed in any form, including as astand-alone program or as a subroutine, element, or other unit suitablefor use in a computing environment. A computer program can be deployedto be executed on one computer or on multiple computers at one or moresites.

FIG. 2 illustrates an exemplary visual program 200 developed in agraphical environment of the VPD environment module 114, in accordancewith some embodiments. As explained herein, a visual program can includeone or more nodes that are connected by one or more interconnections.The exemplary visual program 200 shown in FIG. 2 includes three nodes:image acquisition node 202, image processing node 204, and display node206. As explained herein, each node can include one or more inputterminals for receiving data and one or more output terminals foroutputting data. For example, the image processing node 204 includes oneinput terminal 208 and one output terminal 210.

A node may not have any input terminals. For example, the imageacquisition node 202 only has an output terminal 214, e.g., because thedata processed by the image acquisition node 202 can be internallydefined by the node itself rather than by one or more inputs. Forexample, the image acquisition node 202 can acquire an image using acamera, or the image acquisition node 202 can retrieve data from amemory device. Likewise, a node may not have any output terminals. Forexample, the display node 206 only has an input terminal 216. In thiscase, for example, the data manipulated by the display node 206 can bedisplayed on a display device and/or stored in a memory device. Forexample, the display node 206 can cause the data received at the inputterminal 216 to be display on a display device, such as the displaydevice 108 in FIG. 1. The nodes can be connected by interconnection, forexample, the interconnections 218, 220, to identify a data flow betweenthe nodes.

In some embodiments, the visual program is executed in accordance withthe data flow identified by the nodes and interconnections. For example,when the image acquisition node 202 receives or creates data (e.g.,through a process internally defined in the node 202, such as capturingan image via a camera or a smart camera), the image acquisition node 202can process the data and provide a first output data to an outputterminal 214. The output terminal 214 then provides the first outputdata, through the interconnection 218, to the input terminal of anothernode, such as the input terminal 210 of the image processing node 204.The image processing node 204 can process the first output data receivedat the input terminal 210 (e.g., a captured image) to generate a secondoutput data (e.g., processed image data). The image processing node 204can subsequently provide the second output data to the output terminal212, which is subsequently received by the input terminal 216 of thedisplay node 206. Since the functionality of the visual program isdefined by the functionality of nodes and the interconnections betweenthe nodes, the visual program can visually convey the functionality ofthe program as it is developed.

FIG. 2 shows a simple example of a visual program. One of skill in theart will appreciate that the visual programs can be much more complexthan that shown in FIG. 2, such that they include a large number ofnodes and interconnects. Additionally, other aspects of the visualprogram can be configured, such as whether one or more nodes are to runin parallel with other nodes, if nodes are to execute sequentially(e.g., from left to right as designed in the visual program), and/or thelike. For example, in some embodiments, the transfer of data betweennodes can be asynchronous: a node can transfer data to another node atany time (e.g., once the data is available). In other embodiments, thetransfer of data between nodes can be synchronous. For example, a nodecan transfer data to another node at a predetermined instance of a clockcycle.

FIG. 3 illustrates an exemplary graphical interface of the VPDenvironment module for programming a visual program, in accordance withsome embodiments. The graphical interface 300 can include a programpanel 302 in which a user designs the graphical program by instantiatingvarious node(s) and/or interconnection(s). The graphical interface 300can also include an explorer selection bar 304 with a tree for selectingdevices and/or features for the visual program, including thesub-systems in the VPD environment. Such components in the explorersection bar 304 include scripting blocks (e.g., blocks that allow theuser to create custom scripts), hardware devices (e.g., cameras, smartcameras), recipes (e.g., stores a collection of tag data that can beloaded to customize an application for inspecting similar parts withvarying requirements), a Tag Manager (e.g., a collection of uniqueglobal custom values used to share data among all the sub-systems in theapplication at run-time), Tasks (e.g., a graphical representation of thesequence of interconnected blocks), and Web Pages (e.g., to develop auser interface). For example, the Devices component in the explorerselection bar 304 shows what hardware devices are currently available.For example, the Devices component can display which cameras arephysically connected to the system, and whether the input/output (TO)hardware is recognized and up and running. The Devices component canalso provide access to communication devices, like TCP/IP, Discrete IO,and/or the like.

The VPD can be configured such that when a user selects an element fromthe explorer section, the VPD environment will change to correspond withwhat that sub-system has to offer. Users can select an item in theexplorer selection bar 304 and right click in the program panel 302 tocreate a corresponding block in the graphical program. For example, auser can click on the a Task in the explorer selection bar 304 andright-click in the program panel 302 to create a new Task block for thegraphical program. When the VPD environment creates a new Task block,the toolbox 306 will change to tools appropriate for the newly createdTask. As another example, a user can select the Web Pages in theexplorer selection bar 304 and right click in the program panel 302 tocreate a new Page block for the graphical program. This will change thetoolbox 306 to show tools that are appropriate for a web page.

In some examples, for some components and/or devices in the explorerselection bar 304, a user can double click on a component to modifyassociated properties of the component. For example, if a user selects“camera 0,” the user can double click on it to configure any of theacquisition properties. In some embodiments, if the user double-clickson an item in the explorer selection bar 304, it will bring the user toa corresponding tab 305 in the editor to allow the user to edit theselected item.

The graphical interface 300 can also include a toolbox 306 with agraphical tree for selecting one or more nodes and/or structures for theprogram panel 302. In some embodiments, the toolbox 306 is populatedwith the appropriate tools given what component or sub-system iscurrently being edited in the program panel 302. For example, if a useris editing a Task (e.g., as shown in FIG. 3 by tab 305) then the toolbox306 is populated with tools that are appropriate to be placed into aTask sequence diagram for the graphical program. If a user is editing aweb page (e.g., by selecting the WebPage tab 305B), then the toolbox 306is populated with controls that can be dropped into a web page forcreating a custom HMI.

For example, as shown in FIG. 3 the nodes can include the acquire node308 and the In-Sight Vision Tool Block node 310, which can be selectedfor addition to the program panel 302 using the toolbox 306. The programpanel 302 also includes an interconnect 316 between the output of theacquire node 308 and the input of the InSightToolBlock 310.Additionally, the toolbox 306 can allow the user to perform otherfunctions, such as adding notes. The program panel 302 includes threenotes: note 312A indicates that the visual program is for a singlecamera scenario; note 312B indicates that the acquire node 308 (orblock) performs data acquisition from the camera; and note 312Cindicates that the InSightToolBlock node 310 (or block) performs visionon the camera (e.g., is running the various vision algorithms againstthe acquired or input image). An InSightToolBlock is, for example, anode that represents an executing spreadsheet. It can have inputterminals which are connected to input cells and it can have outputcells that are connected to output terminals, as explained in furtherdetail herein.

As another example, the toolbox 306 allows a user to select a ParallelBlock. In some embodiments, any blocks in the program panel 302 that areinside of the Parallel Block will cause the inside Blocks to be executedin parallel to one another. Without a parallel block, in someembodiments the blocks in a Task Sequence diagram are executed seriallyfrom left to right. So, the parallel block can be used to improve theperformance of the system by causing blocks to run at the same time indifferent threads. For example, using a parallel bloc can result in theoverall Task Sequence from completing faster.

The graphical program interface 300 can also include a properties panel314 for modifying properties of an object selected in the program panel302. In FIG. 3, the note 312C is selected, as indicated by the darkeroutline around the note 312C compared to the outlines of the remainingobjects in the program panel 302. By selecting the note 312C, thegraphical interface allows the user to modify the properties of the note312C, namely setting the comment to “This Block performs vision on thecamera.” The graphical interface 300 can therefore enable a user to add,configure and modify a number of nodes and interconnections between thenodes to create the visual program.

In some embodiments, the graphical interface 300 can enable a user tospecify a computing device for executing an operation associate with anode. For example, when a user selects a node, the user can be providedwith a property pane in which a user can specify the computing devicefor executing that particular selected node.

In some embodiments, the graphical interface 300 can provide aspreadsheet node. A spreadsheet node is a node that is associated with aspreadsheet, such that the node is configured to perform one or moreoperation(s) as defined in the spreadsheet. Thus, a spreadsheet node canbe used to incorporate a spreadsheet functionality into a graphicalprogramming environment. The specific operation of the spreadsheet canbe defined by a user using a spreadsheet interface that is displayed tothe user. U.S. Pat. No. 7,107,519, issued on Sep. 12, 2006, entitledtitled “Spread-Sheet Based User Interface Creation,” assigned to CognexCorporation, describes an exemplary spreadsheet interface forprogramming a spreadsheet, which is incorporated by reference herein inits entirety.

The spreadsheet interface associated with the spreadsheet node candisplay the rows and columns of cells for the spreadsheet. The contentof a cell can be specified by a user during an options specifyingprocess, and may include, for example, the addition or specification offunctions, mathematical expressions, data labels, and/or data to beprocessed according to the desired operation. In addition, input and/oroutput functions can be placed in cells, e.g., where input functionsread information from hardware devices and output functions interactwith specialized hardware devices. Together, the cell expressions in aspreadsheet define the portions of the visual program associated withthe spreadsheet node. Because of the flexibility of the spreadsheetprogram, the spreadsheet interface can be used to specify and/or developcustom features for the machine vision system that are not otherwiseoffered by the graphical programming environment.

In some embodiments, when a user selects a spreadsheet node (e.g., auser double-clicks on a spreadsheet node, such as the InSightToolBlock310 from FIG. 3), the graphical interface of the VPD environment module114 can provide a spreadsheet interface. FIG. 4 illustrates an exampleof a spreadsheet interface 402 shown in the graphical interface 300 ofthe VPD environment module 114, in accordance with some embodiments. Asshown, the spreadsheet interface 402 can include a plurality of cellsarranged in rows and columns. FIG. 4 shows columns A through S and rows0 through 20.

As will be described further herein, the spreadsheet interface 402 canallow a user to configure input(s) and output(s) for the associated nodein the VPD environment. For example, any cell in the spreadsheet can beconfigured as an input or an output for the associated node in theprogram panel 302. For example, a user can configure the spreadsheet tospecify any cell as its Input Image, which the user can process furtherin the spreadsheet (e.g., by defining or using one or more functions inthe spreadsheet). The spreadsheet can allow multiple inputs and/ormultiple outputs. Additionally, as further described herein, thespreadsheet can be configured to publish the value of any cell (e.g., avalue set by processing and/or functions in the spreadsheet) to a HMI toeasily allow binding of HMI controls to cells in a spreadsheet.

The spreadsheet interface can be pre-configured with predeterminedfunctions to aid programming of the spreadsheet. For example, thespreadsheet interface can include the programming functionalitiesdescribed in U.S. Pat. No. 9,123,093, issued on Sep. 1, 2015, entitled“Vision inspection programming method and apparatus,” assigned to CognexCorp, which is incorporated by reference herein in its entirety. Thetoolbox 306 of the graphical interface 300 can be configured to allow auser to select one or more predefined functions for a particular cell ofthe spreadsheet interface 402. For example, as shown in FIG. 4, cell 3Aincludes a DetectBlobs function from the toolbox 306 which, as explainedin the comment section of the toolbox 306, detects blobs from an imageand sorts them by area, returning a blob structure. The user canconfigure the inputs and outputs of the DetectBlobs function byidentifying cells in the spreadsheet interface 402. As another example,the spreadsheet interface can allow a user to add a script (e.g.,JavaScript code) to a cell, such that the script is executed when thespreadsheet executes, as described further herein.

In some embodiments, an input terminal of a spreadsheet node can be usedto provide a cell content to the cell associated with that inputterminal. For example, in FIG. 4, when data is received by thespreadsheet node at the input terminal, the received data is provided asthe content of the cell A0 associated with that input terminal. Thespreadsheet can use the content of the cell A0 to process the data usingthe operation specified in the spreadsheet. For example, functions andoperations can be added to the spreadsheet that use A0 as a parameter.

In some embodiments, an output terminal of a spreadsheet node can beused to retrieve a cell content from the spreadsheet to be used by othernodes in the visual program. For example, in FIG. 4, when a spreadsheetprovides an output of an operation to a cell associated with the outputterminal, the spreadsheet node provides that output to the outputterminal so that other nodes connected to the output terminal of thespreadsheet node can use that output.

In some embodiments, a single spreadsheet can include one or more cellsdesignated as input terminals and/or one or more cells designated asoutput terminals for the spreadsheet node. For example, a user can addentries for inputs and/or outputs to the properties panel 314 byright-clicking on a cell in 402. By designating the cell, the VPDenvironment can, for example, bring up a pop-up context menu that allowsthe user to select a menu item to specify the cell as either an input oran output. In some embodiments, if a cell is designed as input or outputto the spreadsheet, the associated cell will get an adornment (e.g., inthe lower right corner of the cell box, such as a small shading as shownin the table for cells A0 and E9), and the properties panel 314 will beautomatically updated with a pin name and associated information for thepin (e.g., the input or output).

The program panel can display the one or more input terminals and one ormore output terminals for the spreadsheet node. For example, theproperties panel 314 shows the list of inputs and outputs for thespreadsheet. The image at cell A0 is defined as an input to thespreadsheet, and the output of one or more functions created in thespreadsheet interface 402 is populated into the spreadsheet at cell E9as a string data type. Subsequent nodes added to the graphical programcan use the output at E9 by graphically linking to the output of thespreadsheet node (which will be the value populated to cell E9 duringruntime). The ability to define the input(s) and/or output(s) of aparticular spreadsheet node via the cells of the spreadsheet creates thelinkage between the spreadsheet and the graphical programmingenvironment.

The data types for the input(s) and/or output(s) can be pre-defined bythe graphical programming environment. In some examples, the graphicalprogramming environment can allow simple data objects to be used asinputs and outputs, such as digital images, string data types, integerdata types, double data types, and/or the like. In some examples, thegraphical programming environment can support complex data types, suchas a composite of primitive data types that are organized in somefashion. Complex data types could include, for example, lists of (x,y)data points, calibration information (e.g., matrix information,non-linear data), and/or the like.

While FIG. 4 shows only one input and output, FIG. 5 illustrates aspreadsheet node that includes two input terminals and two outputterminals, in accordance with some embodiments. As shown in theproperties panel 314, the spreadsheet includes two inputs: (1) cell A0,which corresponds to an image that is an input to the spreadsheet, and(2) cell A15, which corresponds to a double integer data type that is aninput to the spreadsheet. The spreadsheet also includes two outputs: (1)a string data type that is mapped to cell E9 for output, and (2) adouble data type that is mapped to cell B3 for output.

In some embodiments, the graphical interface can automatically showand/or update the input and/or output terminals attached to thespreadsheet node (e.g., in the program panel 302) based on cellsidentified as an input terminal or an output terminal in the spreadsheetinterface. Thus, creation of the input and output pins can be donethrough the spreadsheet interface 402 by designating a cell as an inputor an output. Once created through the spreadsheet interface 402, thevisual programming development environment can automatically update thedisplay to show input and output pins in the program panel 302. Forexample, suppose a user configures a spreadsheet associated with aspreadsheet node to have one input terminal and one output terminal(e.g., as defined in the property pane 404 of FIG. 4). When a userreturns to the program panel 302, the program panel 302 can graphicallyshow one input terminal and one output terminal coupled to thespreadsheet node (e.g., when otherwise the program panel 302 may show noinputs or outputs prior to configuration).

The spreadsheet interface 402 can provide various techniques forinserting inputs and outputs. For example, a right-click menu can beprovided to allow a user to publish input pins and/or output pins. FIG.9 shows an exemplary menu 900 for publishing an input and/or an outputof a spreadsheet, according to some embodiments. A user can invoke themenu 900 by right-clicking on a cell in the spreadsheet interface. Themenu 900 includes a number of options for a particular cell in thespreadsheet interface to set the cell as an input pin or an output pin,and/or to manipulate the cell. Option Publish to HMI 902 can be selectedto publish the cell to an HMI (e.g., to have the value of a celldisplayed in an HMI, as explained further herein). Option Publish asInput Pin 904 can be selected to publish the cell as an input pin (e.g.,to display an input pin on the corresponding block in the block diagramthat is associated with the cell). Option Publish as Output Pin 906 canbe selected to publish the cell as an output pin (e.g., to display anoutput pin on the corresponding block in the block diagram that isassociated with the cell). Option Unpublish 908 can be selected tounpublish the cell (e.g., to remove a pin displayed on the block).

Each cell in the spreadsheet interface 402 can include one or moreproperties, such as for name, input, and output (e.g., as shown by theproperties panel 314). In some embodiments, when the user publishes acell as either an input or an output, the program chooses a default namefor a pin based on the cell properties and updates the cell's propertiesin the camera. A “changed” event signals both the properties panel 314and the corresponding block in the program panel 302 to update, ifnecessary. For example, if the program determines that the block is anew input or output cell, the program publishes a corresponding pin forthe block in the graphical diagram in the program panel 302.

Once the VPD environment module 114 receives a visual program, the VPDenvironment module 114 can package the visual program as a project fileand provide the project file to the central controller 116.Subsequently, the central controller 116 can unpack the visual program(e.g., traverses the file and generates objects in memory in accordancewith the type specified in the file). In some embodiments, the centralcontroller 116 generates an object for each node, and that object knowshow to execute based upon the parameters of the object. For example, forthe visual program defined in FIG. 2, the central controller 116 candetermine the objects for each of the nodes 202, 204, 206, and schedulethe objects for the nodes 202, 204, 206 to be executed sequentially, asdefined in FIG. 2.

When a node in the visual program is associated with a predeterminedmachine readable program (e.g., the node is a script node), the centralcontroller 116 can simply retrieve that predetermined machine readableprogram. When a node in the visual program is not associated with apredetermined machine readable program (e.g., the node is a spreadsheetnode), the central controller 116 can organize the operation(s) definedby that node into a human readable representation of that portion of theprogram for execution.

For example, the human readable representation of the visual program canbe created using JavaScript Object Notation (JSON), which is anopen-standard format that uses human readable text. FIGS. 10A-10C showan exemplary object for a spreadsheet node, according to someembodiments. FIGS. 10A-10C show a SheetBlock object 1002. The SheetBlockincludes the “SpreadsheetData” entry 1004, which includes the datacontained within the spreadsheet. For illustrative purposes, theplaceholder “<DATA>” was included for the “SpreadsheetData” in place ofactual spreadsheet data. In some embodiments, the spreadsheet data canbe stored in base64 encoded binary format). In some embodiments, thespreadsheet data (e.g., contained within the “SpreadsheetData” entry)includes cells which have names, cells associated with input flags,and/or cells associated with output flags. In some embodiments, thenames of the cells in the spreadsheet data match the NamedCell objects.The SheetBlock object 1002 can also include information that specifiesthe inputs and outputs of the spreadsheet. For example, the NamedCellentry 1004 includes the following information: it is an image(“DataType”: “Image”), and it is an input to the spreadsheet (“IsInput”:true).

FIG. 11 is an exemplary list of objects for cells in a spreadsheet,according to some embodiments. The list includes a list of entries foreach cell that explain the contents of the cell. In this example, thefirst entry 1104 shows that cell A0 of the spreadsheet (pointed to byarrow 1106) contains a function call InputImage (pointed to by arrow1108). The second entry 1110 shows that cell B5 contains a label (shownby “expression”: “Thresh”), and does not contain a function like thatshown in entry 1104.

As shown in the examples in FIGS. 10A-10C and FIG. 11, the centralcontroller has a list of nodes defined by the visual programmingenvironment. The central controller turns each node into an object inmemory. The central controller figures out which node to run based uponleft to right order as they are created (or another order, if defined bythe user) and then executes the objects. For example, if one of thenodes is a spreadsheet, the central controller can hand off that data toa camera for execution. As another example, if one of the nodes is ascript, then the central controller can compile and execute the script.

For example, when the node is a spreadsheet node, the central controller116 can determine the dependencies of the cells. For example, if a cellis an input, then the input needs to be processed before one or morecells can act on the input. As another example, if a cell is an output,then the cell(s) that populate the output value need to be executedbefore returning the output.

In some embodiments, a single spreadsheet node can be used to set up andconfigure a plurality of cameras (e.g., one or more, or all of which areexecuting remotely from the central controller). The cameras can be, forexample, smart cameras that include processors and memory that allow thecamera to execute the spreadsheet and/or functions defined by thespreadsheet. A user can specify the target camera(s) directly in thespreadsheet so that the user can specify the remoteness of the graphicalprogram. The controller (e.g., the central controller 104 shown in FIG.1, such as an embedded controller) can be configured to coordinate theexecution of the spreadsheet across the set of cameras, such as, forexample, passing data between the remote cameras.

In some embodiments, the programming environment generates the list ofoperations that need to take place for the spreadsheet, which can beused to create a dependency graph that represents all the operations andthe data. For example, the controller can pass a block of informationthat represents a spreadsheet to a camera for execution. The camera canopen up the spreadsheet and identify the dependency and order ofoperations it should execute. The camera can process the dependencygraph to determine, for example, where the data is, where it needs togo, and what operation(s) the camera performs.

In some embodiments, the techniques described herein can use the processof serialization to create the human readable file. Serialization caninclude, for example, configuring the cells in the spreadsheet,configuring the input(s) for the spreadsheet, configuring the output(s)for the spreadsheet, and encapsulating the spreadsheet information tosave it to a human readable project file. The host device can send thatproject file down to the controller (e.g., the central controller 104shown in FIG. 1, such as an embedded controller). The central controllercan be configured to coordinate execution of the spreadsheet(s) to oneor more associated cameras, while the functionality defined by the otherblocks of the graphical program can execute on the controller itself.

In some embodiments, the data acquired by a smart camera (e.g., theimage data) can be passed back to the controller in an abbreviated form.For example, since the image data can be large, the system can beconfigured such that rather than copying an image from the camera to thecontroller, the controller can access the images via a URL that accessesimages on the camera by going through the controller's network stack.

In some embodiments, when a node of a visual program is a display node,that node can be configured to display data on a display device 108. Ifthere multiple display devices 108, the display type node can specifythe particular display device 108 on which to display the data.

In some embodiments, a display node may be associated with a particularhuman machine interface (HMI). For example, the HMI can include agraphical interface that is designed to show data on a display device108. In some embodiments, the HMI can be a web-based HMI. The HMI can becustom-designed by a user so that the user can specify the type of datato be displayed on the HMI and the manner in which the data is displayedon the display device 108. The techniques described herein can allowinformation to be passed from the spreadsheet to the HMI. For example,the host device can provide a graphical user interface builder with atool pallet that allows a user to drag buttons/text boxes onto a canvasto graphically design a user interface (the HMI). When a particulargraphical item is dragged onto the canvas, the properties for aparticular graphical item can be configured to point at one or morecells of the spreadsheet. For example, the text for a text box can beconfigured to display the value populated into a particular spreadsheetcell during execution of the spreadsheet.

While FIG. 3 shows an example using a single spreadsheet (associatedwith the InSightToolBlock 310) for a single camera (e.g., the camerathat executes the Acquire node 308), the techniques described herein canuse other configurations including a plurality of cameras and/or aplurality of nodes that support spreadsheets. FIG. 6 shows an exemplaryprogram panel 600 of a graphical interface (e.g., the graphicalinterface 300 in FIG. 3) for a visual program with multiple cameras, inaccordance with some embodiments. The program panel includes an acquirenode 602 configured to perform image acquisition from two differentcameras at the same time, as indicated in comment 604. The imagesacquired from one camera are processed by the InSightToolBlock node 606.The images acquired from the other camera are processed by theInSightToolBlock node 608. Therefore, each InSightToolBlock 606, 608performs vision processing on data from an associated camera used by theacquire node 602, as indicated in comment 610.

FIG. 7 shows an exemplary program panel 700 of a graphical interface fora visual program with a single task with multiple nodes (that each use aspreadsheet) for the same camera, in accordance with some embodiments.The acquire node 702 is configured to perform image acquisition using asingle camera, as explained in the comment 704. The output of theacquire node 702 is input to the InSightToolBlock 706, which isconfigured to perform vision processing on the data acquired by thecamera, as explained in comment 708. The output of the InSightToolBlock706 is fed into InSightToolBlock1 710, which runs after InSightToolBlock706, as explained in the comment 712. Each of the InSightToolBlock 706and InSightToolBlock1 710 are configured using a different, associatedspreadsheet.

FIG. 8 shows an exemplary program panel 800 of a graphical interface fora visual program with multiple nodes (that each use a spreadsheet) withmultiple cameras, according to some embodiments. The acquire node 802acquires image data using four different cameras. The data acquired byeach camera is processed by different nodes. The output of one camera isprocessed by InSightToolBlock 806, and the output of theInSightToolBlock 806 is processed by InSightToolBlock1 808. The outputof a second camera is processed by InSightToolBlock3 810, and the outputof the InSightToolBlock3 810 is processed by InSightToolBlock2 812. Theoutput of a third camera is processed by InSightToolBlock5 814, and theoutput of the InSightToolBlock5 814 is processed by InSightToolBlock4816. The output of a fourth camera is processed by InSightToolBlock7818, and the output of the InSightToolBlock7 818 is processed byInSightToolBlock6 820. Each of the InSightToolBlocks configured using adifferent, associated spreadsheet. The InSightToolBlocks are containedwithin a parallel block 822. Therefore, InSightToolBlocks 806, 810, 814and 818 each simultaneously perform vision calculations on the dataoutput from the respective camera. Similarly, InSightToolBlocks 808,812, 816 and 820 run after the first set of InSightToolBlocks.

It is to be understood that the disclosed subject matter is not limitedin its application to the details of construction and to thearrangements of the components set forth in the following description orillustrated in the drawings. The disclosed subject matter is capable ofother embodiments and of being practiced and carried out in variousways. Also, it is to be understood that the phraseology and terminologyemployed herein are for the purpose of description and should not beregarded as limiting.

As such, those skilled in the art will appreciate that the conception,upon which this disclosure is based, may readily be utilized as a basisfor the designing of other structures, methods, and apparatus forcarrying out the several purposes of the disclosed subject matter. It isimportant, therefore, that the claims be regarded as including suchequivalent constructions insofar as they do not depart from the spiritand scope of the disclosed subject matter. For example, some of thedisclosed embodiments relate one or more variables. This relationshipmay be expressed using a mathematical equation. However, one of ordinaryskill in the art may also express the same relationship between the oneor more variables using a different mathematical equation bytransforming the disclosed mathematical equation. It is important thatthe claims be regarded as including such equivalent relationshipsbetween the one or more variables.

Although the disclosed subject matter has been described and illustratedin the foregoing exemplary embodiments, it is understood that thepresent disclosure has been made only by way of example, and thatnumerous changes in the details of implementation of the disclosedsubject matter may be made without departing from the spirit and scopeof the disclosed subject matter.

1.-22. (canceled)
 23. A computer-implemented method for editing agraphical program using a graphical programming development environment,the method comprising: selecting a spreadsheet node in the graphicalprogramming development environment, wherein: the spreadsheet nodegraphically represents a portion of a graphical program; and thespreadsheet node is associated with a spreadsheet comprising a pluralityof cells arranged graphically in a plurality of rows, a plurality ofcolumns, or both; receiving data indicative of a function for insertioninto a cell of the plurality of cells of the spreadsheet, the functionbeing configured to perform at least a portion of a machine vision task;and generating a computer executable version of the graphical programconfigured to perform the machine vision task, including the functionspecified by the spreadsheet in the spreadsheet node.
 24. The method ofclaim 23, further comprising transmitting the computer executableversion of the graphical program to a controller that is configured toexecute the graphical program.
 25. The method of claim 23, furthercomprising receiving an execution property of the spreadsheet nodeindicating a computing device designated to execute computer-readableinstructions of the spreadsheet node.
 26. The method of claim 25,further comprising: determining, at the controller, based on theexecution property of the spreadsheet node, that the computing device isdesignated to execute the computer-readable instructions of thespreadsheet node; and transmitting the computer-readable instructions ofthe spreadsheet node to the computing device for execution of thecomputer-readable instructions of the spreadsheet node.
 27. The methodof claim 23, further comprising: upon receiving a request to view thespreadsheet associated with the spreadsheet node, displaying thespreadsheet in the graphical programming development environment;receiving second data to be associated with a cell in the spreadsheet;and populating the cell in the spreadsheet using the second data. 28.The method of claim 23, wherein the computing device is a camera. 29.The method of claim 23, wherein the computer-readable instructions ofthe spreadsheet node are designed to perform machine vision analysis ofan image.
 30. The method of claim 23, further comprising providing asecond spreadsheet node in the graphical programming developmentenvironment, wherein the second spreadsheet node is associated with asecond spreadsheet designed to be executed at another computing device.31. The method of claim 23, wherein the spreadsheet node comprises anoutput terminal, wherein the output terminal is associated with anothercell of the plurality of cells in the spreadsheet indicative of anoutput of the computer-readable instructions of the spreadsheet node.32. The method of claim 31, further comprising connecting the outputterminal of the spreadsheet node to a terminal of a node of thegraphical program to provide the output of the computer-readableinstructions of the spreadsheet node to the terminal of the node of thegraphical program.
 33. The method of claim 23, further comprisingproviding a visualization interface having a graphical element, andassociating the graphical element with the cell of the plurality ofcells in the spreadsheet to display a content of the cell at thegraphical element.
 34. The method of claim 33, wherein the content ofthe cell comprises an image, and wherein associating the graphicalelement with the cell comprises assigning, to the graphical element, apointer that references the image.
 35. The method of claim 33, furthercomprising displaying a portion of the spreadsheet in the graphicalelement of the visualization interface.
 36. The method of claim 35,further comprising receiving a request to modify the cell of theplurality of cells of the spreadsheet using the visualization interface.37. The method of claim 23, further comprising: executing thecomputer-readable instructions of the spreadsheet node at a computingdevice, wherein executing the computer-readable instructions comprises:receiving a pointer to second data to be processed by the computingdevice; determining whether the computing device maintains the seconddata locally at the computing device; upon determining that the seconddata is maintained locally at the computing device, processing thesecond data; and upon determining that the second data is not maintainedlocally at the computing device, retrieving the second data from astorage medium referenced by the pointer and processing the second data.38. The method of claim 23, wherein a computing device comprises aplurality of computing modules, and wherein the method further comprisesautomatically selecting one or more of the computing modules to executecomputer-readable instructions of the spreadsheet node.
 39. The methodof claim 38, wherein automatically selecting the one or more of thecomputing modules comprises determining a computation load of the one ormore of the computing modules.
 40. The method of claim 23, wherein thespreadsheet node comprises an input terminal, wherein the input terminalis associated with a cell in the spreadsheet indicative of an inputreceived by the input terminal for processing by a function from thelist of functions.
 41. A system for editing a graphical program using agraphical programming development environment, comprising: a processorin communication with a memory, wherein the processor is configured torun a computer program stored in the memory that is configured to:selecting a spreadsheet node in the graphical programming developmentenvironment, wherein: the spreadsheet node graphically represents aportion of a graphical program; and the spreadsheet node is associatedwith a spreadsheet comprising a plurality of cells arranged graphicallyin a plurality of rows, a plurality of columns, or both; receiving dataindicative of a function for insertion into a cell of the plurality ofcells of the spreadsheet, the function being configured to perform atleast a portion of a machine vision task; and generating a computerexecutable version of the graphical program configured to perform themachine vision task, including the function specified by the spreadsheetin the spreadsheet node.
 42. A non-transitory computer readable mediumhaving executable instructions associated with a system for editing agraphical program using a graphical programming development environment,operable to cause the system to: select a spreadsheet node in thegraphical programming development environment, wherein: the spreadsheetnode graphically represents a portion of a graphical program; and thespreadsheet node is associated with a spreadsheet comprising a pluralityof cells arranged graphically in a plurality of rows, a plurality ofcolumns, or both; receive data indicative of a function for insertioninto a cell of the plurality of cells of the spreadsheet, the functionbeing configured to perform at least a portion of a machine vision task;and generate a computer executable version of the graphical programconfigured to perform the machine vision task, including the functionspecified by the spreadsheet in the spreadsheet node.